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REAL-TIME OBSTACLE DETECTION WITH A CALIBRATED CAMERA AND 

KNOWN EGO-MOTION 



CROSS REFERENCE TO RELATED APPLICATIONS 
This application claims the benefit of U.S. Provisional Application Serial No. 
60/445,203, filed on February 5, 2003, which is incorporated by reference herein in its entirety. 

BACKGROUND OF THE INVENTION 

1, Field of the Invention 

The present invention relates to the field of obstacle detection, and, more particularly, to 
real-time obstacle detection with a calibrated camera and known ego-motion. 

2. Description of the Related Art 

Real-time obstacle detection in a moving vehicle is a key component for autonomous 
driving and computer vision-based driving assistance. Obstacle detection in autonomous driving 
and driving assistance is often used to avoid collision. An obstacle is generally context 
dependent. That is, although an obstacle on the road is usually a vehicle, it can be something 
else, such as a large rock. 

In Serge Beucher et al. Road Monitoring and Obstacle Detection System by Image 
Analysis and Mathematical Morphology, Proceedings 5th EAEC (European Automobile 
Engineers Cooperation) International Congress, "The European Automotive Industry Meets the 
Challenges of the Year 2000", Strasbourg, 21-23 June 1995, March 1995, obstacle detection is 
performed with morphological operations using the road/lane 



detection results. The obstacle is assumed to be a vehicle with two distinct features: (a) it has 
lower dark parts such as a bumper or shadows, and (b) it has a regular rectangular shape. 
Problems with this method include its strong assumption that the obstacle must be a vehicle and 
its dependency on correct road and/or lane detection. 

In Massimo Bertozzi et al. Sensing of Automotive Environments Using Stereo Vision, 
30th Intemational Symposium on Automotive Technology and Automation (ISATA), Special 
Session on Machine Vision and Intelligent Vehicles and Autonomous Robots, pages 187-193, 
Florence, Italy, June 16-19, 1997, an inverse perspective mapping (IPM) is used to obtain a top 
view of the road plane for each of two stereo cameras. The difference of the two remapped 
images shows the high rising obstacles off the road plane. This method requires more than one 
camera, and it cannot detect obstacles with small heights moving on the road. 

In Heinrich Niemann et al.. Integrated motion and geometry based obstacle detection in 
image sequences of traffic scenes, 10th Annual Symposium AeroSense '96, Orlando, FL, 1996, a 
current image is back projected to a road plane in a three-dimensional world system using a 
current transformation between a camera and the real world. Vehicle movement information is 
applied to obtain the location of the road in the previous frame time. Then the road is mapped 
back to the image using the transformation in the previous frame. The registered image with ego- 
motion cancelled out is compared with the previous image to fmd obstacles. This method 
requires warping of the current frame image to a previous frame image with ego-motion 
correcting. This method also requires increased computation power. It also does not use 
information from other sensors in the vehicle, such as the front wheel angle to get more accurate 
vehicle movement status. 
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SUMMARY OF THE INVENTION 
In one aspect of the present invention, a method for real-time obstacle detection from a 
vehicle moving relative to a road is provided^ The method comprises calibrating an image 

5 capturing device, such that any image point can be mapped to a real-world point; receiving one 
or more images from the image capturing device; receiving information about the ego-motion of 
the image capturing device; selecting one or more real-world points according to a tractabiUty 
criterion; applying a motion estimation method on ttie one or more real-world points, the one or 
more images, and the road to determine an image motion flow of the one or more real-world 

10 points; determining a true motion flow of the one or more real-world points based on the one or 
more real world points, the one or more images, and the ego-motion; and determining that the 
one or more real-world points is one or more obstacle points by comparing the image motion 
flow and the true motion flow. 

In another aspect of the present invention, a system for real-time obstacle detection from 

15 a vehicle moving relative to a road is provided. The system comprises an image capturing 
device; means for calibrating an image capturing device, such that any image point can be 
mapped to a real-world point; means for receiving one or more images from the image capturing 
device; means for receiving information about the ego-motion of the image capturing device; 
means for selecting one or more real-world pomts according to a tractability criterion; means for 

20 applying a motion estimation method on the one or more real-world pomts, the one or more 

images, and the road to determine an image motion flow of the one or more real-world points; 
means for determining a true motion flow of the one or more real-world points based on the one 
or more real world points, the one or more images, and the ego-motion; and means for 
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determining that the one or more real-world points is one or more obstacle points by comparing 
the image motion flow and the true motion flow. , 

In yet another aspect of the present invention, a machine-readable medium having 
instructions stored thereon for execution by a processor to perform a method for real-time 

5 obstacle detection from a vehicle moving relative to a road is provided. The method comprises 
calibrating an image capturing device, such that any image point can be mapped to a real-world 
point; receiving one or more images from the image capturing device; receiving information 
about the ego-motion of the image capturing device; selecting one or more real-world points 
according to a tractability criterion; applying a motion estimation method on the one or more 

10 real-world points, the one or more images, and the road to determine an image motion flow of 
the one or more real-world points; determining a true motion flow of the one or more real-world 
points based on the one or more real world pomts, the one or more images, and the ego-motion; 
and determining that the one or more real-^world points is one or more obstacle points by 
comparing the image motion flow and the true motion flow. 

15 

BRIEF DESCRIPTION OF THE DRAWINGS 
The invention may be xmderstood by reference to the following description taken in 
conjunction with the accompanying drawings, in which like reference numerals identify like 
elements, and in which: 

20 Figure 1 shows a block diagram of a processor-based system, in accordance with one 

embodiment of the present invention; 

Figure 2 shows a flow diagram of a method of real-time obstacle detection, in accordiance 
with one embodiment of the present invention; 
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Figure 3 shows a simplified model of a turning vehicle; 

Figure 4(A) shows images captured from an image capturing device, in accordance with 
one embodiment of the present invention; 

Figure 4(B) shows an image motion flow, in accordance with one embodiment of the 
5 present invention; 

Figure 4(C) shows a true motion flow, in accordance with one embodiment of the present 
invention; 

Figure 4(D) shows an output of a comparison between the image motion flow and the 
true motion flow, in accordance with one embodiment of the present invention; and 
10 Figure 5 shows an alternate output of a plurality of comparisons between the image 

motion flows and the true motion flows, in accordance with one embodiment of the present 
invention. 

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS 
15 Illustrative embodiments of the invention are described below. In the interest of clarity, 

not all features of an actual implementation are described in this specification. It will of course 
be appreciated that in the development of any such actual embodiment, numerous 
implementation-specific decisions must be made to achieve the developers' specific goals, such 
as compliance with system-related and business-related constraints, which will vary from one 
20 implementation to another. Moreover, it will be appreciated that such a development effort 
might be complex and time-consuming, but would nevertheless be a routine undertaking for 
those of ordinary skill in the art having the benefit of this disclosure. 
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While the invention is susceptible to various modifications and alternative forms, specific 
embodiments thereof have been shovvn by way of example in the drawings and are herein 
described in detail. It should be understood, however, that the description herein of specific 
embodiments is not intended to limit the invention to the particular forms disclosed, but on the 

5 contrary, the intention is to cover all modifications, equivalents, and alternatives falling within 
the spirit and scope of the invention as defined by the appended claims. 

It is to be understood that the systems and methods described herein may be implemented 
in various forms of hardware, software, firmware, special purpose processors, or a combmation 
thereof In particular, at least a portion of the present invention is preferably implemented as an 

1 0 application comprising program instructions that are tangibly embodied on one or more program 
storage devices (e.g., hard disk, magnetic floppy disk, RAM, ROM, CD ROM, etc.) and 
executable by any device or machine comprising suitable architecture, such as a general purpose 
digital computer having a processor, memory, and input/output interfaces. It is to be further 
understood that, because some of the constituent system components and process steps depicted 

15 in the accompanying Figures are preferably implemented in software, the connections between 
system modules (or the logic flow of method steps) may differ depending upon the manner in 
which the present invention is programmed. Given the teachings herein, one of ordinary skill in 
the related art will be able to contemplate these and similar implementations of the present 
invention. 

20 Referring now to Figure 1 , accordinjg to an embodiment of the present invention, a 

computer system 101 for implementing the present invention can comprise, inter alia, a central 
processing unit (CPU) 102, a memory 103 and an input/output (I/O) interface 104. The 
computer system 101 is operatively coupled through the I/O interface 104 to a display 105 and 
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various input devices 106 such as a mouse and keyboard. The computer system 101 may further 
comprise any of a variety of support circuits (not shown) such as such as cache, power supplies, 
clock circuits, and a communications bus. The memory 103 can include random access memory 
(RAM), read only memory (ROM), disk drive, tape drive, and the like, or a combination thereof 
The present invention can be implemented as a routine 107 that is stored in memory 103 and 
executed by the CPU 102 to process the signal from the signal source 108. As such, the 
computer system 101 is a general purpose computer system that becomes a specific pmpose 
computer system when executing the routine 107 of the present invention. 

The computer system 101 also includes an operating system (not shown) and micro 
instruction code (not shown), which may be stored, for example, in the memory 103. The 
various processes and functions described herein may either be part of the micro instruction code 
or part of the routine 107 (or a combination thereof) which is executed via the operating system. 
In addition, various other peripheral devices (not shown) may be connected to the computer 
system 101 such as an additional data storage device and a printing device. 

The basic principle of at least one embodiment of the present invention is as follows. A 
video camera is installed in a vehicle with known position and orientation. It is imderstood that 
any of a variety of image capturing devices may be used as is contemplated by those skilled in 
the art. The camera is calibrated such that any point in the real world (hereinafter "real-world 
point"), assuming its location is known, can be mapped to a point in the image captured by the 
camera (hereinafter "image point"). The road is assumed to be in an ideal plane and that road 
plane is known. When the vehicle moves, its movement information, such as velocity and 
angular velocity, is available from other sensors onboard. A variety of such sensors are known 
to those skilled in the art, such as a gyro ^d a Global Positioning System (GPS). The movement 
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of the camera, i.e., its ego-motion, can be determined based on the movement information of the 
vehicle. 

Based on the known ego-motion of the camera in reference to the known road plane, any 
real-world point on the road plane has a projected movement, which is commonly known as 
''true motion flow" or "geometric motion flow." The camera captures the actual movement of 
that point. From these camera images, we can compute the movement ofthe image point (/.e., 
the point on the image corresponding to the real-world point). This movement is commonly 
known as "image motion flow." If the movement of the real-world point and the image point 
match, we claim the point is in the road plane. Otherwise, we claim the point is either moving in 
the road plane or not in the road plane. Both are considered to be "obstacle points" indicating 
the potential for an obstacle. It should be noted that it is possible for an out-of-road-plane 
obstacle point to move in such a way that the real-world point and the image point niatch, which 
would falsely indicate that no obstacle is present, but such a situation is rarely seen in real road 
conditions. 

We assume that an obstacle either lies on the road plane or does not lie on the road plane. 
Obstacles lying on a road plane move relative to the road (e.g., a moving car on a road moves 
relative to the road). Obstacles not on a road plane may be static {e.g., the top of an electric pole) 
or moving (e.g. , a low flying bird in the sky). As a result, the motion field of the moving 
obstacle in the captured video would not match that of the road plane. 

It is important to determine which image points and associated image motion flows are 
reliable. The video camera generates images in short intervals periodically. The timing of such 
intervals may be varied, as is understood by those skilled in the art. A motion estimation method 
can reliably compute the displacements of the points (i.e., the image motion flow) from 
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consecutive images. An exemplary motion estimation method is the optical flow technique with 
statistical fusion described in U.S. Patent Application Ser. No. 10/681,702, entitled "Variable- 
Bandwidth Density-Based Fusion", filed Oct. 8, 2003, and assigned to the assignee of the present 
application. The full disclosure of the above-referenced patent application is hereby 

5 incorporated herein by reference. The reliability of points used to determine the image motion 

flow varies due to the nature of the nearby image pattems. For example, the image motion flow 
computed for a point in a homogeneous road patch is far less reliable than that for a comer point 
on a lane marker. The optical flow technique with statistical fusion described above is able to 
distinguish image points with reliable image motion flows from image points with unreliable 

10 flows. The unreliable flows are discarded. Thus, the output of applying the optical flow 
technique described above are the reliable image point locations and their associated image 
motion flows. 

With a calibrated camera, any real-world point location can be mapped to a 
corresponding image point location. An image point location generally cannot be mapped to a 

15 corresponding a real-world point location because the ambiguity along the viewing ray of the 
camera usually cannot be resolved. However, if we know the image point is in a known plane, 
then the latter mapping is possible. Therefore, after the reliable image point locations are 
determined, we can determine the corresponding real-world point locations, assuming that the 
reliable image point locations are in the road plane. 

20 If the reliable image motion flow and the true motion flow of a point match, then the 

point is considered to be fixed on the road plane and not a potential obstacle. If they do not 
match, then the point is considered to be an obstacle point, which indicates a potential obistacle 
either moving in the road plane, or not in the road plane at all. 
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A determination of whether an obstacle point indicates an actual obstacle may be 
determined over a period of time, as is contemplated by those skilled in the art. For example, 
evidence may be collected over a known period of time, and, if a point consistently has two 
mismatching flows, then an obstacle point (i.e., potential obstacle) can be considered to be an 
actual obstacle. 

The method described herein is very fast and reliable. The image motion flow detection 
is based on a fast and reliable optical flow method with statistical fusion. The true motion flow 
calculation is analytical and fast, and it is accurate if the camera calibration, road plane 
information and vehicle sensors are accurate. Further, by verifying a mismatch over a period of 
time, the accuracy of determining whether an obstacle point indicates an actual obstacle 
increases, thus obtaining robust obstacle detection. 

Referring now to Figure 2, a flow diagram 200 of one embodiment of the present 
invention is shown. In performing this method, the following are assumed. First, a road plane is 
known Second, a moving vehicle is on the road plane moving relative to the road plane, and a 
video camera is operatively connected to the vehicle. For example, the video camera can be 
placed on the dashboard of the vehicle to capture images facing the front windshield (e.g., 
similar to what the driver or passenger of the moving vehicle would see). Third, the ego-motion 
of the video camera relative to the road plane is known as well. The ego-motion can be 
determined by using any one or combination of a variety of onboard vehicle sensors, as 
previously discussed. In an alternate embodiment, the ego-motion of the video camera can be 
determined by using a plurality of images gathered by the video camera. 

The video camera with known ego-motion is calibrated (at 205). One or more images 
are gathered (at 210) from the video camera. A real-world point is selected (at 215) according to 
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a tractability criterion. The tractability criterion may be inversely proportional with the 
uncertainty associated with localization of the real-world points. The optical flow technique 
described above is applied on the real-worid point, the one or more images, and the known plane 
to determine (at 220) an image motion flow of the real-worid point. A true motion flow of the 
real-worid point is determined (at 225) from the real-worid point, the one or more images, and 
the known ego motion. The true motion flow and image motion flow are compared to determine 
(at 230) whether the real-world point is an obstacle point. The real-world point may be 
considered an obstacle point if the image motion flow and the true motion flow do not match. 
The real-world point may considered not to be an obstacle point if the image motion flow and the 
true motion flow do match. The obstacle point determines a point of a potential obstacle. Over a 
particular interval of time, as is contemplated by those skilled in the art, if the real-world point is 
consistently determined to be an obstacle point, then the obstacle point may be considered to be 
an actual obstacle. If, over a particular interval of time, the real-world point is not consistently 
determined to be an obstacle point, then the obstacle point may be considered to not be an actual 
obstacle. Other criteria may be used to assist in determining whether an obstacle point is an 
actual obstacle, as is contemplated by those skilled in the art. For example, the close proximity 
of a plurality of obstacle points may further indicate the presence of an obstacle. 

A theoretical analysis of the present invention, in accordance with one embodiment of the 
present invention, will now be described. The mapping of locations is as follows. The 
transformation between the world coordinates and the camera frame coordinates is: 

P^^R{P^-T) (1) 

where, = {X^.Y^.Z^Y is the world coordinates, T is the displacement between origins of 
the two coordinate systems, R is the 3x3 rotation matrix between orientations of the two 
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coordinate systems, and = {X^,Y^,ZJ^ is the point in the camera frame. It should be noted 
that R effectively has only three free elements: the roll (rotation around the X axis), the pitch 
(rotation around the Y axis) and the yaw (rotation around the Z axis). R and T arc camera 
extrinsic parameters, which map world coordinates into the camera reference frame. 

We assume flie image frame, camera frame and world frame have similar structure, Le., 
the X axis points forward, the Y axis points to the left, and the Z axis points upward. 
According to a canonical camera model, tiie image coordinates (y,,Z,)^ are constrained by: 
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-(Y,-oJs=f 



(2) 



-(Z,-oj5,=/ 



X. 



(3) 



where 0^,0^,5^,5^ and / are camera intrinsic parameters, which map the camera reference 

frame coordinates in the image plane. 
If we define 



the pixel coordinates are: 
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By plugging Eq. (1) into Eq. (4), we get: 
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where RJ is the /th row of the R matrix. This is the transformation of any world point to the 
image plane. 

If the world points are on a plane, they satisfy the plane equation: 
Since we can choose the origin to avoid ^/ = 0 , we use a normalized plane equation, 
The Eq. (1) can be written as: 

P^=RF^-RT = RP^''RTA'^P,=R(I-TA^)P^ 



10 By pluggmg Eq. ( 1 0) into Eq. (4), we get: 







1 0 


0 




-{- 


0 


<^3) 




0. 0 


_£ 



R(I^TA')P^ 



If we define 
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then 
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(13) 



H is the homography matrix to map a world plane to the image plane. We define the world 
coordinate system in this way: the origin is a fixed point in the vehicle and the orientation is 
fixed in regard to the vehicle, e.g., the X axis pomts to the front of the vehicle, the Y axis points 
to the left of the vehicle, and the Z axis points to the top of the vehicle. For clarity, we will call 
this tiie "vehicle coordinate system" if necessary in the following context, and the subscript will 
change to v instead of w . We fiirther install the camera so that it is fixed on the vehicle. 
Therefore, R and T are fixed. If we assume the intrinsic camera parameters, 
Oy,o^-flSy -fls^ do not change, and the normalized plane equation (/.e., Eq. (9)) holds in the 

vehicle coordinate system (/.e., A does not change), then H is constant. 

For the last requirement that A does not change, the origin of the vehicle coordinate 
system can either move in a straight line or tum around the normal vector of the road plane in a 
plane parallel to the road plane. This restriction is reasonable for very smooth roads. Slow 
slopes of the road makes H only approximately correct. Quick slopes or vehicle rollovers 
violate the requirement. In the following, we will assume H is fixed over the time. 

On the other hand, if we know the point in the image plane, we should be able to get the 
point in the world plane. H is normally not singular," or the camera would actually see the road 
plane as a line in the image, which is avoided in a real system setting. Then: 
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Because A^P^=\, 



y 



(15) 



So: 



R. =■ 
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(16) 



Therefore, if we assume the image pixels correspond to points on the world plane, Eq. (16) gives 
the world point on flie plane. 

The mapping of velocities is as follow. The velocity of the world point in the road plane 
can be expressed as the derivative in regard to time t : 



F = 



. 7' 



dt 

dt 
dZ, 



(17) 



Let us define Hj as the ith row of H , then 



X3 - HjP^ 



(18) 
(19) 
(20) 



Now we need to establish the mapping of the image velocity and world velocity: 
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Similarly, if we know the image point and its velocity in the image plane, we can take the 
derivative of Eq. (16) to get the velocity of the point in the world. 
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(23) 



Previously, we established the mappings of the locations and velocities in the vehicle and 
the image coordinate systems. We still need to find out how the world moves viewing from the 
vehicle, i.e., the movement of the world in the vehicle coordinate system. 

When the vehicle moves in a straight line at a known velocity V^ ^ , each world point 

moves at P'^ = -V^^^ in the vehicle coordinate system. Then for a pixel in the image plane, its 
correspondent world location in the vehicle coordinate system, , can be computed through Eq. 
(16). Fiuther, its true motion flow in the image can be computed readily from and P\ by use 
Eq.(21)andEq.(22). 

However, when the vehicle makes a turn, the velocity of the world point in the vehicle 
coordinate system is no longer homogenous. Let us consider pure in-plane rotations, /.e., the 
vehicle tums when it runs on the perfect plane and no skidding occurs. According to Figure 3, 
the vehicle is simplified to two wheels: the (average) front wheel and the (average) rear wheel. 
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The term "average" is used in the following sense. A four wheel vehicle is simplified to have 
two wheels only: the "average" front wheel at the center point between the two front wheels and 
the "average" rear wheel at the center point between the two rear wheels. The vehicle 
movement information is given according to the simplified vehicle model, e.g., the (average) 
front wheel speed, and the (average) front wheel angle. With known front wheel angle S and 
wheel axle L , the front wheel turning radius is: 

L 



R = 



sin^ 



(25) 



If the physical front wheel speed V^^ is known, the angular speed is V^^ I R = Vf^s.mS I L . We 



denote fl? as the angular velocity vector: 



0= 



f \ 
0 
0 

^ L 



(26) 



For any point on the vehicle, its velocity in regard to the rotation O is: 



(27) 



Because the vehicle coordinate system is fixed with the vehicle orientation in regard to its 
origin , any non-vehicle world point P„ moves with a velocity 

VMP.-Oyco (28) 



= (/'^-0,-0)x<y 
= (P,-OJxfi;-n,„ 



(29) 
(30) 
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where V^^ is the velocity of the origin of the vehicle coordinate system in the real world. Notice 
that - is actually the world point in the vehicle coordinate system: So, the velocity of a 

non-vehicle world point in thie vehicle world system is 

K-^P^xco-K,, (31) 

The non-rotational movement is a special case where fi;= 0 , so 

K--K^. (32) 
With Eq. (31), the velocity of the world point is known for turning cases, so its true 
image motion flow can easily be computed from Eq. (21) and Eq. (22). 

We assume the origin is a point on the axle between the centers of the (average) front 
wheel and (average) rear wheel, with a distance pL to the center of the rear wheel. Similar to 
S , the angle between the velocity of the origin and the X axis, )5 , can be computed as 

)ff = arctan(/7tanJ) . (33) 



The (scalar) speed of the vehicle origin is = Vj-^yjcos^ S+p^ sin^ S , so. 
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(34) 



(35) 



For the special case of straight line movement, S = 0 and = 0 , then 
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v = 



0 



V 



0 



(36) 



If we simply put the origin of the vehicle coordinate system on the front wheel center, 
then jt? = 1 and /? = J . Note that we need to change H because we use a different T . So, 



V =V 



r Y ^ 
— ^sin^-cos<y 
L 

sin<y-sin<y 

L 

0 



(37) 



Based on the theoretical analysis described above, the workings of the present invention 
will now be described, in accordance with one embodiment of the present invention. 

We install the camera fixed on the vehicle, determine the vehicle coordinate system by 
putting the origin fixed on a point in the vehicle. Therefore, we know the rotation R and 
displacement T from the vehicle coordinate system to the camera frame. The relationship of the 
road plane to the vehicle coordinate system is also determined, /.e., A is established. Due to the 

way we select the vehicle coordinate system, A has the form of (0,0, c)^ . Then, the 
homography H is easily computed from the camera intrinsic parameters, R , T and A , 
according to Eq. (12). 

When we receive an image in the video from the camera, we can compute the image 
motion flow for selected points with high confidence, using sophisticated algorithms described 
elsewhere. At the same time, from what we know about the point, the location in the image 
plane and z , we can compute the world point using Eq. (16). Eq. (35) or Eq. (37) gives 



F,. the velocity of a world point in the vehicle coordinate system. So, plugging and P\=V^ 
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into Eq. (21) and Eq, (22), we get the true image motion flow of the point. Comparing the 
measured image motion flow with the computed true motion flow (which assimies it is a point on 
the road plane), we can check if the point is really a road point. As mentioned before, an 
obstacle point's image motion flow does not match the true motion flow computed assuming it is 
a road point, so the obstacle is detected. 

Referring now to Figure 4, an application of the method of Figure 2 is illustrated, in 
accordance with one embodiment of the present invention. Figure 4(a) shows two images 405, 
410 captured from a calibrated video camera with known ego-motion relative to a known road 
plane. Second image 410 represents an elapsed time interval after first image 405. Although 
only two images 405, 410 are shown for the sake of simplicity, it is understood that any number 
of images may be gathered, as is contemplated by those skilled ui the art. The two images 405, 
410 may be sent to an image motion flow detection unit to determine the image motion flow. 

Figure 4(B) shows an exemplary image motion flow 415 with one reliable moving 
pattem 420. The reliable moving pattern 420 refers to the tip of a lane marker. More 
specifically, the reliable moving pattem 420 represents the tip of the lane marker nioving from 
the first image 405 to the second image 410. Although only one reliable moving pattem 405 is 
shown, it is understood that the image motion flow detection unit may determine numerous 
reliable moving patterns 420 within the image motion flow. The image motion flow may be sent 
to a true motion flow detection unit. 

Figure 4(C) shows an exemplary true motion flow 425 with two road point movements: a 
lane marker movement 430 and a rear wheel movement 435. The two road point movements 
430, 435 represent projected movements of the lane marker and the rear wheel, respectively. As 
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previously described, these are determined using the known ego-motion of the camera. The 
image motion flow and the true motion flow may be sent to a comparison unit. 

Figure 4(D) shows an exemplary output 440 after comparing the image motion flow and 
the true motion flow. The reliable moving pattem 420 of Figure 4(B) matches the lane marker 
movement 430 of Figure 4(C); therefore, they are considered to not be an obstacle point. The 
rear wheel movement 435 of Figure 4(C), on the other hand, does not have a respective match in 
the image motion flow .415 of Figure 4(B). Therefore, the rear wheel movement 435 is 
represented in the output 440 as an obstacle point 445. In one embodiment, the obstacle point 
445 may be determined after one iteration of the comparison unit. In an altemate embodiment, 
the obstacle point 445 may be determined after a plurality of iterations of the comparison unit to 
assure that the rear wheel movement 435 has no match in the image motion flow 415. Figure 5 
shows an altemate exemplary ouQ)ut 450 with a plurality of obstacle points 445. 

It is understood that the present invention may be combined with any of a variety of 
known obstacle detection methods and systems. Such combinations may provide greater 
accuracy and/or more robust methods and systems. For example, the present invention may be 
combined with a radar, lidar, and ultrasonic obstacle detection systems and methods. 
Furthermore, once an obstacle point is determined to be an actual obstacle, the type of obstacle , 
may be classified. For example, any of a variety of known classification methods may be used. 
It may be implemented using an automated classification system and/or a graphical user 
interface. The graphical user interface may allow human input to add greater accuracy to the 
classification process. 

The particular embodiments disclosed above are illustrative only, as the invention may be 
modified and practiced in different but equivalent manners apparent to those skilled in the art 
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having the benefit of the teachings herem. Furthermore, no limitations are intended to the details 
of construction or design herein shown, other than as described in the claims below. It is 
therefore evident that the particular embodiments disclosed above may be altered or modified 
and all such variations are considered wittiin the scope and spirit of the invention. Accordingly, 
the protection sought herein is as set forth in the claims below. 
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